Apparatus and method for reducing image blur in a digital camera

ABSTRACT

A digital camera delays the capture of a digital image after image capture has been requested until the motion of the digital camera satisfies a motion criterion. The digital camera thereby reduces image blur that would otherwise occur due to camera motion.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of application Ser. No. 10/339,132filed on Jan. 8, 2003, now U.S. Pat. No. 7,212,230 which is herebyincorporated by reference herein.

FIELD OF THE INVENTION

The present invention relates generally to digital photography and morespecifically to apparatuses and techniques for reducing image blur in adigital camera.

BACKGROUND OF THE INVENTION

A pervasive problem in photography is blur due to camera motion. Somefilm cameras and other optical devices such as binoculars include highlysophisticated active image stabilization systems that deflect the imagepath slightly in a direction opposite of the camera motion. Such activestabilization systems are, however, both complex and expensive.

One alternative is to use a faster lens. Digital cameras already use thefastest lens practical in terms of cost, size, and desired imagequality. Lenses with maximum apertures of f/2 to f/2.8 are typical.Still faster lenses are much more expensive and bulky.

It is thus apparent that there is a need in the art for a digital camerathat reduces image blur without recourse to expensive or otherwiseimpractical solutions.

SUMMARY OF THE INVENTION

A method for reducing image blur in a digital camera by the tracking ofcamera motion is provided. An apparatus for carrying out the method isalso provided.

Other aspects and advantages of the present invention will becomeapparent from the following detailed description, taken in conjunctionwith the accompanying drawings, illustrating by way of example theprinciples of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a functional block diagram of a digital camera in accordancewith an illustrative embodiment of the invention.

FIG. 1B is a conceptual diagram of the motion management logic shown in

FIG. 1A in accordance with an illustrative embodiment of the invention.

FIG. 1C is a circuit diagram of an input control in accordance with anillustrative embodiment of the invention.

FIG. 2A is an illustration of a digital preview frame, of which acentral portion is a motion measurement region, in accordance with anillustrative embodiment of the invention.

FIG. 2B is an illustration of a digital preview frame, of which aperipheral portion is a motion measurement region, in accordance with anillustrative embodiment of the invention

FIG. 2C is an illustration of separate horizontal and vertical sets ofpicture elements that may be used in measuring the motion of a digitalcamera in accordance with an illustrative embodiment of the invention.

FIG. 3A is an illustration of a movement trajectory of a digital camerain accordance with an illustrative embodiment of the invention.

FIG. 3B is an illustrative plot of the magnitude of composite cameramotion annotated with time and threshold parameters in accordance withan illustrative embodiment of the invention.

FIG. 4 is a flowchart of the operation of the digital camera shown inFIG. 1A in accordance with an illustrative embodiment of the invention.

FIG. 5 is a flowchart of the operation of the digital camera shown inFIG. 1A in accordance with another illustrative embodiment of theinvention.

DETAILED DESCRIPTION OF THE INVENTION

Camera motion or “shake” tends to be somewhat periodic in both thehorizontal and vertical directions. At instants of direction reversal,the camera is nearly stationary, just as a child's swing is momentarilystationary at the instant it reaches the full extent of its travel ineither direction. Image blur in a digital camera may therefore bereduced by delaying capture of a digital image, after actuation of theshutter release button, until the motion of the digital camera satisfiesa motion criterion. For example, image capture may be delayed untilcamera motion reaches one of those instants of minimal motion (a localminimum). To avoid an unacceptable lag following actuation of theshutter release button, the delay may be constrained not to exceed apredetermined timeout period, or other criteria may be used to capture adigital image despite the motion criterion not being satisfied.Implementation of this technique requires some method for measuring themotion of the digital camera in approximately real time. Motionestimation algorithms may be relatively simple or quite complex. Oneexample of sophisticated motion estimation well known in the videoencoding art is that implemented in connection with the Moving PicturesExpert Group (MPEG) video compression standards.

FIG. 1A is a block diagram of a digital camera 100 in accordance with anillustrative embodiment of the invention. In FIG. 1A, controller 105communicates over data bus 110 with imaging module 115, input control120, display 125, motion management logic 130, timer 135, and memory140. Memory 140 further comprises random-access memory (RAM) 145 andnon-volatile memory 150. Optical system 155 produces optical images thatare converted to digital images by imaging module 115. Optical system155 may comprise, for example, a zoom lens. Imaging module 115 maycomprise an array of photosensors based on charge-coupled-device (CCD)or CMOS technology, an analog-to-digital converter (A/D), a gaincontrol, and a digital signal processor (DSP) (not shown in FIG. 1A).Imaging module 115 may be operated in a video preview mode in whichdigital preview frames are acquired at a rate of, for example, 30 framesper second and shown on display 125. Digital camera 100 may operate inthis video preview mode while operations such as autofocus,autoexposure, and motion tracking are performed. A CMOS photosensorarray has the advantage that pixels can be addressed directly like RAM,which simplifies and speeds the readout of the image data for theseoperations.

Motion management logic 130 may comprise hardware, firmware, software,or a combination thereof. Motion management logic 130 may beconceptualized as having two aspects: motion measurement logic 132 andcontrol logic 134, as shown in FIG. 1B. Motion measurement logic 132performs motion estimation on digital preview frames obtained fromimaging module 115 during video preview mode. These digital previewframes may be of a lower resolution than a final digital image tofacilitate video preview mode. Control logic 134 analyzes motionestimation information obtained from motion measurement logic 132 todetermine when a digital image should be captured after capture of animage has been requested. In one illustrative embodiment, controller 105comprises a microprocessor, and motion management logic 130 comprisesstored program instructions in software or firmware or a combinationthereof that may be executed by controller 105. In such an illustrativeembodiment, the combination of controller 105, imaging module 115, andmotion measurement logic 132 may be termed, functionally, a motiontracking subsystem that outputs an indication of the motion of digitalcamera 100 as a function of time. Controller 105, in accordance withcontrol logic 134, selects the instant of image capture based on theoutput of the motion tracking subsystem.

FIG. 1C is a circuit diagram of input control 120 in accordance with anillustrative embodiment of the invention. In FIG. 1C, shutter releasebutton 160 is capable of actuating, sequentially, switches S1 165 and S2170. When shutter release button 160 is partially depressed, switch S1165 is closed. When shutter release button 160 is further depressed,switch S2 170 is also closed. Prior to the closing of switches S1 165and S2 170, signals 175 and 180, respectively, are both in a logic“high” state. Signals 175 and 180 are connected with data bus 110. Thelogic “high” state is provided by connection of the switches between acommon ground and a positive voltage +V across pull-up resistors 185 and190. When switch S1 165 is closed, the corresponding signal 175 ispulled down to ground potential, generating a logic “low.” Likewise,when switch S2 170 is closed, the corresponding signal 180 is pulleddown to ground potential, generating a logic “low.”

Input control 120 may be used to trigger multiple operations in digitalcamera 100. For example, actuation of switch S1 165 may activateautofocus and autoexposure. Once autofocus and autoexposure adjustmentsare complete, a motion-tracking mode may be activated in which themotion of digital camera 100 is tracked. Actuation of S2 170 may signala request that a digital image be captured and stored. In a prior-artdigital camera, such capture would be immediate (without intentionaldelay). To minimize image blur caused by camera motion, however, it isadvantageous to delay capture of the digital image until a moment whenthe motion of digital camera 100 is at an approximate local minimum. Inother embodiments, input control 120 may include only one switch insteadof two. In those embodiments, a single signal from input control 120 mayrequest the capture of a digital image, and motion tracking may beactivated by a separate input signal (e.g., the power of digital camera100 being turned on) or by the same single signal from input control120.

Measuring the motion of digital camera 100 may be implemented in avariety of ways. The most obvious is through the use of motion sensors,such as accelerometers or gyroscopes. In better keeping with the lowcost and complexity objectives of the invention, motion can be measuredusing the image sensor itself. Motion estimation algorithms are wellknown in the video encoding art. Motion estimation generally involvescomparing at least one picture element (pixel) in a first frame with atleast one pixel in a second frame to discern a change in the sceneduring the interval between the two frames. This process may be repeatedfor successive pairs of frames to track camera motion relative to thebackground of the scene in approximately real time. In the context ofthe instant invention, motion estimation may be performed on digitalpreview frames obtained in the video preview mode of digital camera 100.

The comparison of pixels may also be implemented in a variety of ways.For example, the magnitude of the pixel-by-pixel difference inbrightness (luminance) may be computed. Alternatively, a pixel-by-pixelcorrelation (multiplication) may be performed. If the pixels comparedare in corresponding locations in the two digital preview frames, anindication may be obtained that motion of some sort between the framesoccurred but not how much or in what direction. For this reason, motionestimation techniques typically also include a search algorithm in whichone or more groups of pixels in a first digital preview frame arecompared with groups of pixels within a predetermined search regionsurrounding each corresponding location in a second digital previewframe. A motion estimation algorithm typically computes a motion vectorindicating the magnitude and direction of motion during a particularinterval. This motion vector may be expressed conveniently as horizontaland vertical motion components.

Sophisticated motion estimation techniques used in connection with MPEGcompression may improve the performance of motion estimation. Suchimprovements may include, for example, a fast search algorithm or anefficient computational scheme in addition to the general methoddescribed above. Such methods are well known in the video encoding art.One example of sophisticated MPEG motion estimation may be found in U.S.Pat. No. 6,480,629, the disclosure of which is incorporated herein byreference.

FIG. 2A depicts a digital preview frame 205 in accordance with anillustrative embodiment of the invention. Motion estimation may beperformed using one or more pixels within motion measurement region 210(cross-hatched in FIG. 2A). In FIG. 2A, motion measurement region 210comprises a central portion of digital preview frame 205. Such a regionmay coincide with the region used in performing autofocus orautoexposure. In such an embodiment, motion estimation may sharevirtually the same video preview mode of digital camera 100 withautofocus and autoexposure. One disadvantage of this approach, however,is that a moving subject within the central portion of digital previewframe 205 may be detected instead of the motion of digital camera 100relative to the background.

FIG. 2B shows one method for overcoming the problem of subject motion inaccordance with an illustrative embodiment of the invention. In FIG. 2B,motion measurement region 210 comprises a peripheral portion of digitalpreview frame 205 where an important subject is less likely to be found.By confining motion estimation to the periphery, subject motion may beexcluded, allowing the motion of digital camera 100 relative to thebackground of the scene to be measured.

In performing autofocus, digital cameras often apply a window functionat the boundary of the autofocus region to minimize edge effects causedby contrasty image data at the boundary. The window function attenuatesthe edges of the autofocus region in a tapered fashion, resulting in a“soft” boundary. Such window functions are well known in the digitalcamera art. A window function applied at the boundary 215 delineatingmotion measurement region 210 may be advantageous for the same reason.

FIG. 2C is an illustration of separate horizontal and vertical sets ofpixels that may be used in performing motion estimation in accordancewith an illustrative embodiment of the invention. In FIG. 2C, horizontalsets of pixels 220 and vertical sets of pixels 225, both lying withinperipheral motion measurement region 210, may be used in performingmotion estimation, as described above. Horizontal and vertical sets ofpixels 220 and 225, respectively, may be single rows or columns ofpixels or “strips” of pixels comprising multiple rows or columns. Oneadvantage of this approach is that measurement of horizontal andvertical motion may be separated into two sets of computations (e.g.,difference or correlation), each set of computations producing anestimated motion component. The limited number of pixels involved mayalso simplify the search algorithm. The choice of pixel sets shown inFIG. 2C is only one possibility of many. Fewer or more than the foursets of horizontal and vertical sets of pixels shown in FIG. 2C may beincluded in motion estimation.

FIG. 3A is an illustration of a movement trajectory of a digital camerain accordance with an illustrative embodiment of the invention.Trajectory 305 in FIG. 3A depicts the path of movement made by digitalcamera 100 during an arbitrary period prior to an image being captured.Local minima 310, where the motion of digital camera 100 changesdirection, are circled. Capturing a digital image at one of these localminima 310 may reduce image blur. Since the motion of digital camera 100may not reach a local minimum in the horizontal direction at the sameinstance it reaches a local minimum in the vertical direction, this mustbe taken into account in designing criteria for image capture.

Many possible quantities may be chosen as the output of the motiontracking subsystem. If motion measurement logic 132 measures horizontaland vertical motion components (e.g., velocities), one possible choiceis the square root of the sum of the horizontal motion component squaredand the vertical motion component squared (magnitude of the motionvector). Another possible choice is the sum of the absolute value of thehorizontal motion component and the absolute value of the verticalmotion component.

FIG. 3B is an illustrative plot 315 of the magnitude of composite motionof digital camera 100 annotated with time and threshold parameters inaccordance with an illustrative embodiment of the invention. In plot315, threshold 320 serves as a motion criterion for selecting theinstant of image capture at an approximate local minimum. In oneembodiment, a digital image is captured when the magnitude of motiondrops below threshold 320 (point 330 in FIG. 3B) following activation ofswitch S2 170 at time 325. If the magnitude of motion does not dropbelow threshold 320 within a predetermined timeout period 335, thedigital image may be captured at time 340. The choice of timeout period335 may vary with application or situation, but it would likely notexceed 0.1 second. As an alternative to timeout period 335, the imagemay be captured, despite threshold 320 not being satisfied, if themotion of digital camera 100 is found to be decreasing (in terms ofvelocity, digital camera 100 is decelerating). An example of such aninterval of decreasing motion is that from point 345 to point 350.Anticipating an approximate minimum in this way is particularly usefulin accounting for readout and computational lags in the motionestimation process.

Threshold 320 may be selected based on any of a variety of factors or acombination thereof. In one embodiment, threshold 320 is selected basedon the minimum and maximum motion measured (e.g., minimum and maximumvelocities) during an interval 355 between actuation of switch S1 165and actuation of switch S2 170 (see FIG. 3B). Such minimum and maximummotion measurements within interval 355 are illustrated by points 360and 365, respectively. Threshold 320 may be chosen, for example, as aparticular fraction of maximum 365 or as a value lying between minimum360 and maximum 365. In a different embodiment, threshold 320 may bechosen based on the current focal length setting of optical system 155.A wide-angle focal length of optical system 155 is less sensitive tomotion than a telephoto setting. Therefore, threshold 320 may need to besmaller for a telephoto focal length than for a wide-angle focal length.Likewise, a faster shutter speed also renders motion of digital camera100 less critical. Consequently, threshold 320 may be larger (lessstrict) if digital camera 100 is operating at a fast shutter speed(e.g., 1/500 of a second) than if digital camera is operating at aslower shutter speed (e.g., 1/30 of a second). Another factor that maybe used in selecting threshold 320 is interval 355 in FIG. 3B. Forexample, threshold 320 may be chosen differently depending on whetherinterval 355 is short or long. In yet another embodiment, the thresholdcan be set based on prior characterization of the typical amount ofcamera motion after actuation of S1 165. For example, if a high degreeof motion is detected, due to an unsteady user, one-handed operation,etc., a higher threshold may be selected.

Optionally, threshold 320 may be altered after actuation of switch S2170. For example, threshold 320 may be increased (making the motioncriterion less strict) with the passage of time after the actuation ofS2. This is yet another alternative to timeout period 335 and capturingthe digital image upon detected deceleration of digital camera 100.

FIG. 4 is a flowchart of the operation of digital camera 100 inaccordance with an illustrative embodiment of the invention. If switchS1 165 is actuated at 405, autofocus and autoexposure are performed at410. Once autofocus and autoexposure are complete, digital camera 100enters a motion measurement mode at 415 in which the motion trackingsubsystem measures the motion of digital camera 100 as a function oftime, as explained above. If switch S2 170 is actuated at 420, controlproceeds to 425, and timer 135 may be reset to count timeout period 335.At 425, the output of motion measurement logic 132 is compared withthreshold 320. If the measured motion is less than threshold 320, adigital image may be captured immediately at 435. Otherwise, controlproceeds to 430. At 430, controller 105 checks timer 135 to determinewhether timeout period 335 has expired. If so, the digital image may becaptured at 435. Otherwise, control returns to 425. Once the digitalimage has been captured, the process terminates at 440.

FIG. 5 is a flowchart of the operation of digital camera 100 inaccordance with another illustrative embodiment of the invention. Theprocess in FIG. 5 is similar to that in FIG. 4, except that a decrease(deceleration, in terms of velocity) in the motion of digital camera 100is the criterion for capturing a digital image at 445, when threshold320 is not satisfied at 425.

The foregoing description of the present invention has been presentedfor the purposes of illustration and description. It is not intended tobe exhaustive or to limit the invention to the precise form disclosed,and other modifications and variations may be possible in light of theabove teachings. The embodiment was chosen and described in order tobest explain the principles of the invention and its practicalapplication to thereby enable others skilled in the art to best utilizethe invention in various embodiments and various modifications as aresuited to the particular use contemplated. It is intended that theappended claims be construed to include other alternative embodiments ofthe invention except insofar as limited by the prior art.

1. A digital camera, comprising: an imaging module configured to convertoptical images to digital images and further configured to produce aseries of digital preview frames in a video preview mode; an inputcontrol to initiate capture of a digital image by the imaging module;and processing logic configured to determine motion of the digitalcamera, the processing logic being configured to delay capture of thedigital image until the determined motion of the digital camerasatisfies a motion criterion, and the processing logic being configuredto perform an automated image capture adjustment operation; wherein theperformance of the automated image capture adjustment operation and thedetermination of the motion of the digital camera by the processinglogic are respectively based on picture element values in differentrespective mutually exclusive regions of each of respective ones of thedigital preview frames, wherein for each of the respective ones of thedigital preview frames, picture element values from a central windowregion of the digital preview frame are used for performing theautomated image capture adjustment operation and picture element valuesfrom a peripheral motion measurement region of the digital preview frameoutside the central window region are used for determining the motion ofthe digital camera.
 2. The digital camera of claim 1, wherein theprocessing logic is configured to apply a window function thatattenuates an edge sub-region of the peripheral motion measurementregion of each of the respective ones of the digital preview frames. 3.The digital camera of claim 1, wherein the processing logic isconfigured to determine a horizontal component of the motion of thedigital camera based on a horizontally oriented set of picture elementsin the peripheral motion measurement region of each of the respectiveones of the digital preview frames.
 4. The digital camera of claim 1,wherein the processing logic is configured to determine a verticalcomponent of the motion of the digital camera based on a verticallyoriented set of picture elements in the peripheral motion measurementregion of each of the respective ones of the digital preview frames. 5.The digital camera of claim 1, wherein the processing logic isconfigured to compute horizontal and vertical motion components of themotion of the digital camera from different respective sets of pictureelements in the peripheral motion measurement region of each of therespective ones of the digital preview frames.
 6. The digital camera ofclaim 1, further comprising at least one accelerometer.
 7. The digitalcamera of claim 1, further comprising at least one gyroscope.
 8. Thedigital camera of claim 1, wherein the processing logic is configured todetermine the motion of the digital camera based on the square root ofthe sum of a horizontal motion component squared and a vertical motioncomponent squared.
 9. The digital camera of claim 1, wherein theprocessing logic is configured to determine the motion of the digitalcamera based on the sum of the absolute value of a horizontal motioncomponent and the absolute value of a vertical motion component.
 10. Thedigital camera of claim 1, wherein the automated image captureadjustment operation that the control logic is configured to perform isan autofocus operation, and the central window region corresponds to anautofocus window.
 11. The digital camera of claim 1, wherein theautomated image capture adjustment operation that the control logic isconfigured to perform is an autoexposure operation, and the centralwindow region corresponds to an autoexposure window.
 12. The digitalcamera of claim 1, wherein in the process of performing the automatedimage capture adjustment operation, the processing logic determines therespective central window sub-region of each of respective ones of thedigital preview frames.
 13. A method for reducing image blur in adigital camera, comprising: acquiring a series of digital preview framesin a video preview mode of the digital camera: determining motion of thedigital camera in response to a first input signal: and delaying captureof a digital image, following receipt of a second input signal, untilthe determined motion of the digital camera satisfies a motioncriterion; and performing an automated image capture adjustmentoperation; wherein the performing of the automated image captureadjustment operation and the determining of the motion of the digitalcamera are respectively based on picture element values in differentrespective mutually exclusive regions of each of respective ones of thedigital preview frames, wherein for each of the respective ones of thedigital preview frames, picture element values from a central windowregion of the digital preview frame are used for performing theautomated image capture adjustment operation and picture element valuesfrom a peripheral motion measurement region of the digital preview frameoutside the central window region are used for measuring the motion ofthe digital.
 14. The method of claim 13, wherein the performingcomprises performing an autofocus operation, and the central windowregion coincides with an autofocus region of each digital preview framewith respect to which the autofocus operation is performed.
 15. Themethod of claim 13, further comprising: applying a window function thatattenuates an edge sub-region of the peripheral motion measurementregion of each of the respective ones of the digital preview frames. 16.The method of claim 13, wherein the determining comprises determining ahorizontal component of the motion of the digital camera based on ahorizontally oriented set of picture elements in the peripheral motionmeasurement region of each of the respective ones of the digital previewframes.
 17. The method of claim 13, wherein the determining comprisesdetermining a vertical component of the motion of the digital camerabased on a vertically oriented set of picture elements in the peripheralmotion measurement region of each of the respective ones of the digitalpreview frames.
 18. The method of claim 13, wherein the determiningcomprises computing horizontal and vertical motion components of themotion of the digital camera from different respective sets of pictureelements in the peripheral motion measurement region of each of therespective ones of the digital preview frames.
 19. The method of claim13, further comprising measuring the motion of the digital camera usingat least one accelerometer.
 20. The method of claim 13, furthercomprising measuring the motion of the digital camera using at least onegyroscope.
 21. A digital camera comprising: means for converting opticalimages to digital images and for producing a series of digital previewframes in a video preview mode; means for initiating capture of adigital image; means for tracking motion of the digital cameraresponsive to the means for initiating capture of a digital image, anoutput of the means for tracking motion of the digital camera indicatingthe quantity of motion of the digital camera as a function of time; andmeans for delaying capture of the digital image until the output of themeans for tracking motion of the digital camera satisfies a motioncriterion; and means for performing an automated image captureadjustment operation; wherein the performing means performs theautomated image capture adjustment operation and the tracking meanstracks the motion of the digital camera based on picture element valuesin different respective mutually exclusive regions of each of respectiveones of the digital preview frames, wherein for each of the respectiveones of the digital preview frames, picture element values from acentral window region of the digital preview frame are used by theperforming means for performing the automated image capture adjustmentoperation and picture element values from a peripheral motionmeasurement region of the digital preview frame outside the centralwindow region are used by the tracking means for tracking the motion ofthe digital camera.